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Abstract 



A satisfactory multiagent learning algorithm 
should, at a minimum, learn to play opti- 
mally against stationary opponents and con- 
verge to a Nash equilibrium in self-play. The 
algorithm that has come closest, WoLF-IGA, 
has been proven to have these two proper- 
ties in 2-player 2-action repeated games — 
assuming that the opponent's (mixed) strat- 
egy is observable. In this paper we present 
AWESOME, the first algorithm that is guar- 
anteed to have these two properties in all re- 
peated (finite) games. It requires only that 
the other players' actual actions (not their 
strategies) can be observed at each step. It 
also learns to play optimally against oppo- 
nents that eventually become stationary. The 
basic idea behind AWESOME (Adapt When 
Everybody is Stationary, Otherwise Move to 
Equilibrium) is to try to adapt to the oth- 
ers' strategies when they appear stationary, 
but otherwise to retreat to a precomputed 
equilibrium strategy. The techniques used to 
prove the properties of AWESOME are fun- 
damentally different from those used for pre- 
vious algorithms, and may help in analyzing 
other multiagent learning algorithms also. 1 



x We thank the anonymous reviewers, as well as Michael 
Bowling and Manuela Veloso for helpful discussions. This 
material is based upon work supported by the Na- 
tional Science Foundation under CAREER Award IRI- 
9703122, Grant IIS-9800994, ITR IIS-0081246, and ITR 
IIS-0121678. 



1. Introduction 

Learning from experience is a key capability in AI, be- 
cause it can be difficult to program a system in advance 
to act appropriately. Learning is especially important 
in multiagent settings where the other agents' behav- 
ior is not known in advance. Multiagent learning is 
complicated by the fact that the other agents may be 
learning as well, thus making the environment nonsta- 
tionary for a learner. 

Multiagent learning has been studied with different 
objectives and different restrictions on the game and 
on what the learner can observe (e.g., (Tan, 1993; Sen 
& Weiss, 1998)). Two minimal desirable properties of 
a good multiagent learning algorithm are 

• Learning to play optimally against stationary op- 
ponents (or even opponents that eventually become 
stationary). 2 

• Convergence to a Nash equilibrium in self-play (that 
is, when all the agents use the same learning algo- 
rithm) . 

These desiderata are minimal in the sense that any 
multiagent learning algorithm that fails at least one 
of these properties is, in a sense, unsatisfactory. Of 
course, one might also want the algorithm to have ad- 
ditional properties. 3 

2 This property has sometimes been called rational- 
ity (Bowling & Veloso, 2002), but we avoid that term be- 
cause it has an established, different meaning in economics. 

3 It can be argued that the two properties are not even 
strong enough to constitute a "minimal" set of require- 
ments, in the sense that we would still not necessarily be 
satisfied with an algorithm if it has these properties. How- 
ever, we would likely not be satisfied with any algorithm 
that did not meet these two requirements, even if it had 
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However, to date there has been no algorithm that 
achieves both of these minimal properties in general 
repeated games. Many of the proposed algorithms 
satisfy the first property (e.g. (Vrieze, 1987; Claus & 
Boutilier, 1998; Singh et al., 2000; Bowling & Vcloso, 
2002; Wang & Sandholm, 2002)). Some of the algo- 
rithms satisfy the second property in restricted games 
(e.g. (Vrieze, 1987; Littman, 1994; Hu & Wellman, 
1998; Singh ct al., 2000; Bowling & Veloso, 2002; Wang 
& Sandholm, 2002)). 

The algorithm that has come closest to satisfying both 
of the properties in general repeated games is WoLF- 
IGA (Bowling & Veloso, 2002). (This algorithm set 
out to do exactly this and was an improvement over 
an earlier algorithm (Singh et al., 2000).) It is guar- 
anteed to have both of the properties in general games 
under the following assumptions: (a) there are at most 
2 players, (b) each player has at most 2 actions to 
choose from, (c) the opponent's strategy (distribution 
over actions) is observable, and (d) gradient ascent of 
infinitesimally small step sizes can be used. 4 

Another learning algorithm that succeeds in achiev- 
ing similar goals is "regret matching" , with which the 
learner's regrets converge to zero and, if all players 
use the learning algorithm, the empirical distributions 
of play converge to a correlated equilibrium (Hart & 
Mas-Colell, 2000). (The set of correlated equilibria is a 
strict superset of the set of Nash equilibria, where play- 
ers are allowed to condition their action on a commonly 
observed signal. Thus, convergence to a Nash equilib- 
rium is a strictly stronger property than convergence 
to a correlated equilibrium.) Convergence to corre- 
lated equilibria is achieved by a number of other learn- 
ing procedures (Cahn, 2000; Foster & Vohra, 1997; 
Fudenberg & Levine, 1999). 

In this paper we present AWESOME, the first algo- 
rithm that has both of the desirable properties in gen- 
eral repeated games. 5 It removes all of the assump- 
tions (a)-(d). It has the two desirable properties with 
any finite number of agents and any finite number of 
actions; it only requires being able to observe other 
players' actions (rather than the distribution that the 
actions are drawn from); and it docs not rely on in- 
finitesimal updates. 

other properties. This is the sense in which we use the 
word "minimal". 

4 Bowling and Veloso also defined a more generally ap- 
plicable algorithm based on the same idea, but only gave 
experimental justification for it. 

5 As in WoLF-IGA, our notion of convergence is that the 
stage-game strategy converges to the desired strategy (not 
just the long-term empirical distribution). 



AWESOME still makes some of the same assumptions 
that were made in the prior theoretical work attempt- 
ing to attain both of the desirable properties (Singh 
et al., 2000; Bowling & Vcloso, 2002). First, (for now) 
it only deals with repeated games — that is, stochas- 
tic games with a single state. Second, it assumes that 
the structure of the game is known (has already been 
learned). Third, as in (Bowling & Veloso, 2002), we 
also assume that the agents can compute a Nash equi- 
librium. 6 (It is still unknown whether a Nash equilib- 
rium can be found in worst-case polynomial time (Pa- 
padimitriou, 2001), but it is known that certain related 
questions are hard in the worst case (Conitzer & Sand- 
holm, 2003). ) 7 

The basic idea behind AWESOME (Adapt When Ev- 
erybody is Stationary, Otherwise Move to Equilibrium) 
is to try to adapt to the other agents' strategies when 
they appear stationary, but otherwise to retreat to a 
precomputed equilibrium strategy. At any point in 
time, AWESOME maintains either of two null hy- 
potheses: that the others are playing the precom- 
puted equilibrium, or that the others are stationary. 
Whenever both of these hypotheses are rejected, AWE- 
SOME restarts completely. AWESOME may reject 
either of these hypotheses based on actions played in 
an epoch. Over time, the epoch length is carefully in- 
creased and the criterion for hypothesis rejection tight- 
ened to obtain the convergence guarantee. The AWE- 
SOME algorithm is also self-aware: when it detects 
that its own actions signal nonstationarity to the oth- 
ers, it restarts itself for synchronization purposes. 

The techniques used in proving the properties of AWE- 
SOME are fundamentally different from those used 
for previous algorithms, because the requirement that 
the opponents' whole strategies can be observed is 
dropped. These techniques may also be valuable in 
the analysis of other learning algorithms in games. 

It is important to emphasize that, when attempting 
to converge to an equilibrium, as is common in the 
literature, our goal is to eventually learn the equilib- 
rium of the one-shot game, which, when played repeat- 
edly, will also constitute an equilibrium of the repeated 

6 We assume that when there are multiple AWESOME 
players, they compute the same Nash equilibrium. This is 
natural since they share the same algorithm. 

7 Much of the literature on learning in games is con- 
cerned with learning (the relevant aspects of) the game it- 
self, or, even when the game is already known, with reach- 
ing the equilibrium through some simple dynamics (not 
using a separate algorithm to compute it). These are cer- 
tainly worthwile objectives in our opinion. However, in this 
paper the uncertainty stems from the opponent, and the 
goal is to play appropriately with respect to the opponent's 
algorithm. 



game. The advantage of such equilibria is that they 
are natural and simple, always exist, and are robust to 
changes in the discounting/averaging schemes. Never- 
theless, in repeated games it is possible to also have 
equilibria that are fundamentally different from repe- 
titions of the one-shot equilibrium; such equilibria rely 
on a player conditioning its future behavior on the op- 
ponents' current behavior. Interestingly, a recent pa- 
per shows that when players are interested in their 
average payoffs, such equilibria can be constructed in 
worst-case polynomial time (Littman & Stone, 2003). 

The rest of the paper is organized as follows. In Sec- 
tion 2, we define the setting. In Section 3, we motivate 
and define the AWESOME algorithm and show how to 
set its parameters soundly. In Section 4, we show that 
AWESOME converges to a best response against op- 
ponents that (eventually) play stationary strategies. 
In Section 5, we show that AWESOME converges to a 
Nash equilibrium in self-play. In Sections 6 and 7, we 
present conclusions and directions for future research. 

2. Model and definitions 

We study multiagent learning in a setting where a fixed 
finite number of agents play the same finite stage game 
repeatedly. We first define the stage game and then 
the repeated game. 

2.1. The stage game 

Definition 1 (Stage game) A stage game is defined 
by a finite set of agents {1,2, ... ,n}, and for each 
agent i, a finite action set A; t , and a utility function 
Ui : A\ x A 2 x . . . x A n — > IR. The agents choose their 
actions independently and concurrently. 

We now define strategies for a stage game. 

Definition 2 (Strategy) A strategy for agent i (in 
the stage game) is a probability distribution iti over its 
action set Ai, indicating what the probability is that 
the agent will play each action. In a pure strategy, all 
the probability mass is on one action. Strategies that 
are not pure are called mixed strategies. 

The agents' strategies are said to be in equilibrium if 
no agent is motivated to unilaterally change its strat- 
egy given the others' strategies: 

Definition 3 (Nash equilibrium (NE)) A strat- 
egy profile (nl , ir^, . . . , 7T*) is a Nash equilibrium (NE) 
(of the stage game) if, for every agent i and for any 



strategy 7r i; 

E(tt* ,...,7r*_ 1 ,7r^' ,7r* +1 ,7r| ,...,7r* ) u i ( a l > a 2 > • ■ • > a n) ^ 

-^(V* ,...,7T*_ 1 ,7Ti,7r* +1 ,7r| )^i(^l ) ^2 ) • • • ) O n ) 

We call a NE a pure-strategy NE if all the individuals ' 
strategies in it are pure. Otherwise, we call it a mixed- 
strategy NE. 

As in most of the game theory literature on learning 
(for a review, see (Fudenberg & Levine, 1998)) and 
in both of the theoretical results on multiagent learn- 
ing in computer science that we are trying to improve 
upon (Bowling & Veloso, 2002; Singh et al., 2000), we 
assume that the agents know the game. So, they do 
not need to learn what the game is, but rather they 
just need to learn how to play. 8 

2.2. The repeated game 

The agents play the stage game repeatedly (forever). 
As usual, we assume that the agents observe each oth- 
ers' actions. An agent may learn from previous rounds, 
so its strategy in a stage game may depend on how the 
earlier stage games have been played. 

In the next section we present our learning algorithm 
for this setting, which has the desirable properties that 
it learns a best response strategy againts opponents 
that eventually are stationary, and it converges to a 
Nash equilibrium in self-play. 

3. The AWESOME algorithm 

In this section we present the AWESOME algorithm. 
We first give the high-level idea, and discuss some ad- 
ditional specifications and their motivation. We then 
give the actual algorithm and the space of valid pa- 
rameter vectors for it. 

3.1. The high-level idea 

Roughly, the idea of the algorithm is the following. 
When the others appear to be playing stationary 
strategies, AWESOME adapts to play the best re- 
sponse to those apparent strategies. When the others 
appear to be adapting their strategies, AWESOME re- 
treats to an equilibrium strategy. (Hence, AWESOME 
stands for Adapt When Everybody is Stationary, Oth- 
erwise Move to Equilibrium.) 

8 If the game were not known and the agents are using 
the same learning algorithm, they could explore the game 
and learn the game structure, and then learn how to play. 
This is also an active area of research in multiagent systems 
in computer science (e.g., (Littman, 1994; Hu & Wellman, 
1998; Claus & Boutilier, 1998; Wang & Sandholm, 2002)). 



3.2. Additional specifications 

While the basic idea is simple, we need a few more 
technical specifications to enable us to prove the de- 
sired properties. 

• To make the algorithm well-specified, we need to 
specify which equilibrium strategy AWESOME re- 
treats to. We let AWESOME compute an equilibrium 
in the beginning, and it will retreat to its strategy in 
that equilibrium every time it retreats. To obtain our 
guarantee of convergence in self-play, we also specify 
that each AWESOME agent computes the same equi- 
librium (this is reasonable since they share the same 
algorithm) . We observe that any equilibrium will work 
here (e.g., a social welfare maximizing one), but AWE- 
SOME might not converge to that equilibrium in self- 
play. 

• We specify that when retreating to the equilibrium 
strategy, AWESOME forgets everything it has learned. 
So, retreating to an equilibrium is a complete restart. 
(This may be wasteful in practice, but makes the anal- 
ysis easier.) 

• To avoid nonconvergence in self-play situations 
where best-responding to strategies that are not quite 
the precomputed equilibrium strategies would lead to 
rapid divergence from the equilibrium, AWESOME at 
various stages has a null hypothesis that the others 
are playing the precomputed equilibrium. AWESOME 
will not reject this hypothesis unless presented with 
significant evidence to the contrary. 

• AWESOME rejects the equilibrium hypothesis also 
when its own actions, chosen according to its mixed 
equilibrium strategy, happen to appear to indicate a 
nonequilibrium strategy (even though the underlying 
mixed strategy is actually the equilibrium strategy). 
This will help in proving convergence in self-play by 
making the learning process synchronized across all 
AWESOME players. (Since the other AWESOME 
players will restart when they detect such nonstation- 
arity, this agent restarts itself to stay synchronized 
with the others.) 

• After AWESOME rejects the equilibrium hypothesis, 
it randomly picks an action and changes its strategy to 
always playing this action. At the end of an epoch, if 
another action would perform significantly better than 
this action against the strategies the others appeared 
to play in the last epoch, it switches to this action. 
(The significant difference is necessary to prevent the 
AWESOME player from switching back and forth be- 
tween multiple best responses to the actual strategies 
played.) 



• Because the others' strategies are unobservable (only 
their actions are observable), we need to specify how 
an AWESOME agent can reject, based on others' ac- 
tions, the hypothesis that the others are playing the 
precomputed equilibrium strategies. Furthermore, we 
need to specify how an AWESOME agent can reject, 
based on others' actions, the hypothesis that the oth- 
ers are drawing their actions according to stationary 
(mixed) strategies. We present these specifications in 
the next subsection. 

3.3. Verifying whether others are playing the 
precomputed equilibrium and detecting 
nonstationarity 

Let us now discuss the problem of how to reject, 
based on observing the others' actions, the hypothesis 
that the others are playing according to the precom- 
puted equilibrium strategies. AWESOME proceeds in 
epochs: at the end of each epoch, for each agent i in 
turn (including itself), it compares the actual distribu- 
tion, hi, of the actions that i played in the epoch (i.e. 
what percentage of the time each action was played) 
against the (mixed) strategy 7r* from the precomputed 
equilibrium. AWESOME concludes that the actions 
are drawn from the equilibrium strategy if and only 
if the distance between the two distributions is small: 
max^gAi \p^. — < e e , where is the percentage 
of time that action a is played in <f>. 

When detecting whether or not an agent is play- 
ing a stationary (potentially mixed) strategy, AWE- 
SOME uses the same idea, except that in the close- 
ness measure, in place of 7r* it uses the actual dis- 
tribution, hf rev , of actions played in the epoch just 
preceding the epoch that just ended. Also, a differ- 
ent threshold may be used: e s in place of e e . So, 
AWESOME maintains the stationarity hypothesis if 
and only max„ i£l 4, \p a h \ ~ Ph» rev \ < e "- 

The naive implementation of this keeps the number of 
iterations N in each epoch constant, as well as e e and 
e s . Two problems are associated with this naive ap- 
proach. First, even if the actions are actually drawn 
from the equilibrium distribution (or a stationary dis- 
tribution when we are trying to ascertain stationarity) , 
there is a fixed nonzero probability that the actions 
taken in any given epoch, by chance, do not appear to 
be drawn from the equilibrium distribution (or, when 
ascertaining stationarity, that the actual distributions 
of actions played in consecutive epochs do not look 
alike). 9 Thus, with probability 1, AWESOME would 

9 This holds for all distributions except those that cor- 
respond to pure strategies. 



eventually restart. So, AWESOME could never con- 
verge (because it will play a random action between 
each pair of restarts). Second, AWESOME would not 
be able to distinguish a strategy from the precomputed 
equilibrium strategy if those strategies are within e e of 
each other. (Similarly, AWESOME would not be able 
to detect nonstationarity if the distributions of actions 
played in consecutive epochs are within e s .) 

We can fix both these problems by letting the distance 
e e and e s decrease each epoch, while simultaneously 
increasing the epoch length N. If we increase N suffi- 
ciently fast, the probability that the equilibrium distri- 
bution would by chance produce a sequence of actions 
that does not appear to be drawn from it will decrease 
each epoch in spite of the decrease in e e . (Similarly, 
the probability that a stationary distribution will, in 
consecutive epochs, produce action distributions that 
are further than e s apart will decrease in spite of the 
decrease in e s .) In fact, these probabilities can be de- 
creased so fast that there is nonzero probability that 
the equilibrium hypothesis (resp. stationarity hypoth- 
esis) will never be rejected over an infinite number 
of epochs. Chebyshev's inequality, which states that 
P(\X - E(X)\ >t)< Var J X) , will be a crucial tool in 
demonstrating this. 

3.4. The algorithm skeleton 

We now present the backbone of the algorithm for re- 
peated games. 

First we describe the variables used in the algorithm. 
Me refers to the AWESOME player, ir* is player p's 
equilibrium strategy. <j> is the AWESOME player's cur- 
rent strategy. h^~ ev and h c v urr are the histories of ac- 
tions played by player p in the previous epoch and the 
epoch just played, respectively. (h™jJe is the vector 
of all h c p urr besides the AWESOME player's.) t is the 
current epoch (reset to every restart). APPE (all 
players playing equilibrium) is true if the equilibrium 
hypothesis has not been rejected. APS (all players sta- 
tionary) is true if the stationarity hypothesis has not 
been rejected. S is true if the equilibrium hypothesis 
was just rejected (and gives one round to adapt before 
the stationarity hypothesis can be rejected), e* , e* , TV* 
are the values of those variables for epoch t. n is the 
number of players, \A\ the maximum number of ac- 
tions for a single player, /j, (also a constant) the utility 
difference between the AWESOME player's best and 
worst outcomes in the game. 

Now the functions. ComputcEquilibriumStrategy 
computes the equilibrium strategy for a player. Play 
takes a strategy as input, and plays an action drawn 
from that distribution. Distance computes the dis- 



tance (as defined above) between strategies (or his- 
tories). V computes the expected utility of playing a 
given strategy or action against a given strategy profile 
for the others. 

We are now ready to present the algorithm. 

AWESOME() 

1 . for each p 

2. 7r* := ComputcEquilibriumStrategy(p) 

3. repeat {// beginning of each restart 

4. for each player p { 

5. InitializeToEmpty(/iP re, 

6. InitializeToEmpty(/i™ rr ) } 

7. APPE := true 

8. APS := true 

9. 5 := false 

10. t := 

11. <f>:= Tr* Me 

12. while APS { // beginning of each epoch 

13. repeat N t times { 

14. Play(0) 

15. for each player p 

16. Update(/i™ rr ) } 

17. if APPE = false { 

18. if S = false 

19. for each player p 

20. if (Distance(/i™ rr , hP p rev ) > e*) 

21. APS := false 

22. 5 := false 

23. a := argmax V(a, h c ^ e ) 

24. if V(a, h™£ e ) > Vf>, h™£ e ) +n|A|e*+V 

25. := a } 

26. if APPE = true 

27. for each player p 

28. if (Distance(/i™"\ ir* p ) > e*) { 

29. APPE := false 

30. 4> :— RandomAction() 

31. S := true } 

32. for each player p { 

22 j^prev ._ focurr 

34. InitializeToEmpty(^ tlrr ) } 

35. t:=t + l}} 

We still need to discuss how to set the schedule for 
(e*, e*, N*). This is the topic of the next section. 

3.5. Valid schedules 

We now need to consider more precisely what good 
schedules are for changing the epochs' parameters. It 
turns out that the following conditions on the schedule 
for decreasing e e and e s while increasing N are suffi- 
cient for the desirable properties to hold. The basic 
idea is to make N go to infinity relatively fast com- 
pared to the e e and e s . The reason for this exact def- 
inition will become clear from the proofs in the next 
section. 



Definition 4 A schedule {(el, e*, iV*)} tg { ,i.2,...} * s 
valid if 

• e* , e* decrease monotonically and converge to 0. 

• iV* -»• oo. 

i 



lte{i,2,...}( 1_ l^ls ^4(^+1)2, 
OJ, w/iere zs ifte total number of actions summed 
over all players. 



> (with all factors > 



• Ilte{i.2,...}( 1 - N t( e t)a 
>0). 
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The next theorem shows that a valid schedule always 
exists. 

Theorem 1 A valid schedule always exists. 



Proof: Let {e* 



e * }te{o,i,2,...} be any de- 



creasing sequence going to 0. Then let N* = 
(which indeed goes to infinity). 
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2St€{i,2,...} (J) _ Because the 
sum in the exponent converges, it follows that this is 
positive. ■ 



4. AWESOME learns a best-response 
against eventually stationary 
opponents 

In this section we show that if the other agents use 
fixed (potentially mixed) strategies, then AWESOME 
learns to play a best-response strategy against the op- 
ponents. This holds even if the opponents are non- 
stationary first (e.g., because they are learning them- 
selves) as long as they become stationary at some time. 

Theorem 2 With a valid schedule, if all the other 
players play fixed strategies forever after some round, 
AWESOME converges to a best response with proba- 
bility 1. 

Proof: We prove this in two parts. First, we prove 
that after any given restart, with nonzero probability, 
the AWESOME player never restarts again. Second, 
we show that after any given restart, the probability of 
never restarting again without converging on the best 
response is 0. It follows that with probability 1, we 
will eventually converge. 



To show that after any given restart, with nonzero 
probability, the AWESOME player never restarts 
again: consider the probability that for all t (t 
being set to right after the restart), we have 

max P1 t 1 {d((t> t p , <j> p )} < (where the AWESOME 

player is player 1, 4> p is the distribution of ac- 
tions actually played by p in epoch t, and <f) p 
is the (stationary) distribution that p is actu- 
ally playing from). This probability is given by 

e t+i 

IltGU^...}! 1 - p { max P^i{d(4> P ^p)} > -V)), which 
is greater than Ilte{i,2,...} i 1 - E P ^i p {d{4> P ,(pp) > 
- S 2~)), which in turn is greater than n*e{i 2 }(1 — 

E P ^iE^(l^(«) - > 4r)) ( whcrc <i>M ^ 

the probability <fi p places on a). Because E(<p P (a)) = 
<f> p (a), and observing Var(<f) p (a)) < we can 

now apply Chebyshev's inequality and conclude that 
the whole product is greater than Ilte-fi 2 } 1 — 
l+i\2 1 where \A\^ is the total number of ac- 

tions summed over all players. 10 But for a valid sched- 
ule, this is greater than 0. 

Now we show that if this event occurs, then APS will 
not be set to false on account of the stationary play- 
ers. This is because d(<j> P , 



u p 



) > e* => d{<j) p ,<j)p) 



dtf- 1 ,^) > e* =*■ d(cb p ,cb p ) > 4f V dO^T 1 ,^) > 

t d {4> P A P ) > V v di^p^Ap) > t ( usin s thc 
triangle inequality and the fact that the e s are stricly 
decreasing). 

All that is left to show for this part is that, given that 
this happens, APS will, with some nonzero probabil- 
ity, not be set to false on account of the AWESOME 
player. Certainly this will not be the case if APPE 
remains true forever, so we can assume that this is set 
to false at some point. Then, with probability at least 
pn, the first action b that thc AWESOME player will 
choose after APPE is set to false is a best response to 
the stationary strategies. (We are making use of the 
fact that the stationary players' actions are indepen- 
dent of this choice.) We now claim that if this occurs, 
then APS will not be set to false on account of the 
AWESOME player, because the AWESOME player 
will play b forever. This is because the expected util- 
ity of playing any action a against players who play 
from distributions cb t >1 (call this Ui (a, </>>i)) can be 
shown to differ at most n\A\ max p ^i d((j) p , <j) P )n from 
the expected utility of playing action a against players 
who play from distributions 0>i (call this u\(a, </>>i)). 
Thus, for any t and any a, we have u\(a, <^>i) < 

10 We observe that we used the fact that the schedule is 
valid to assume that the factors are greater than in the 
manipulation. 



u 1 {a,(j) >1 )+n\A\e t + 1 ^ < ui (6, </>>i) + n\A\e\ +l n (be- 
cause & is a best- response to </>>i), and it follows that 
the AWESOME player will never change its strategy. 

Now, to show that after any given restart, the proba- 
bility of never restarting again without converging on 
the best response is 0: there are two ways in which this 
could happen, namely with APPE being set to true 
forever, or with it set to false at some point. In the 
first case, we can assume that the stationary players 
are not actually playing the precomputed equilibrium 
(because in this case, the AWESOME player would ac- 
tually be best-responding forever). Let p ^ 1 and a be 
such that 4> p {a) ^ Tr*(a), where 7r*(a) is the equilib- 
rium probability p places on a. Let d = \4> p (a)—ir*(a)\. 
By Chebyshev's inequality, the probability that 4> P (a) 
is within | of 4>p{ a ) is a t least 1 — j^p, which goes 
to 1 as t goes to infinity (because N* goes to infin- 
ity). Because e* goes to 0, at some point e* < |, so 
14(a) - cj> p (a)\ < I => W p {a) - K* p {a)\ > c*. With 
probability 1, this will be true for some <f> P (a), and at 
this point APPE will be set to false. So the first 
case happens with probability 0. For the second case 
where APPE is set to false at some point, we can as- 
sume that the AWESOME player is not playing any 
best-response b forever from some point onwards, be- 
cause in this case the AWESOME player would have 
converged on a best response. All we have to show 
is that from any epoch t onwards, with probability 1, 
the AWESOME player will eventually switch actions 
(because starting at some epoch t, e s will be small 
enough that this will cause APS to be set to false). If 
playing an action a against the true profile (/>>i gives 
expected utility k less than playing 6, then by conti- 
nuity, for some e, for any strategy profile (f>' >1 within 
distance e of the true profile 4>>\, playing a against 
4>' >1 gives expected utility at least | less than playing 
b. By an argument similar to that made in the first 
case, the probability of (j) t >1 being within e of the true 
profile (f> > i goes to 1 as t goes to infinity; and because 
eventually, n\A\el +1 fi will be smaller than |, this will 
cause the AWESOME player to change actions. ■ 

5. AWESOME converges to a Nash 
equilibrium in self-play 

In this section we show that AWESOME converges to 
a Nash equilibrium when all the other players are using 
AWESOME as well. 

Theorem 3 With a valid schedule, AWESOME con- 
verges to a Nash equilibrium in self-play with probabil- 
ity 1. 



Proof: We first observe that the values of APPE and 
APS are always the same for all the (AWESOME) 
players, due to the synchronization efforts in the al- 
gorithm. It can be shown in a manner similar to the 
proof of Theorem 2 that after any restart, with nonzero 
probability, we have, for all t, max p {d((f> P ,n*)} < e* 
(where 4> p is the distribution of actions actually played 
by p in epoch t, and 7r* is the equilibrium strategy for 
p). In this case, APPE is never set to false and the 
players play the equilibrium forever. 

All that is left to show is that, after any restart, the 
probability of never restarting while not converging to 
an equilibrium is 0. This can only happen if APPE is 
set to false at some point, and the players do not keep 
playing a pure-strategy equilibrium forever starting at 
some point after this. As in the proof of Theorem 2, 
all we have to show is that from any epoch t onwards, 
with probability 1, some player will eventually switch 
actions (because starting at some epoch t, e s will be 
small enough that this will cause APS to be set to 
false). Because we can assume that at least one player 
is not best-responding to the others' actions, the proof 
of this claim is exactly identical to that given in the 
proof of Theorem 2. ■ 

It is interesting to observe that even in self-play, it is 
possible (with nonzero probability) that AWESOME 
players converge to an equilibrium other than the pre- 
computed equilibrium. Consider a game with a pure- 
strategy equilibrium as well as a mixed-strategy equi- 
librium where every action is played with positive 
probability. If the mixed-strategy equilibrium is the 
one that is precomputed, it is possible that the equilib- 
rium hypothesis (by chance) is rejected, and that each 
player (by chance) picks its pure-strategy action after 
this. Because from here on, the players will always 
be best-responding to what the others are doing, they 
will never change their strategies, the stationarity hy- 
pothesis will never be rejected, and we have converged 
on the pure-strategy equilibrium. 

6. Conclusions 

A satisfactory multiagent learning algorithm should, 
at a minimum, learn to play optimally against sta- 
tionary opponents, and converge to a Nash equilib- 
rium in self-play Surprisingly, current algorithms, 
even those that specifically pursued this pair of prop- 
erties as a goal, do not have these properties. The 
algorithm that has come closest is WoLF-IGA. It has 
been proven to have these two properties in simple 2- 
player 2-action games — further making the unrealistic 
assumptions that the opponent's strategy is known at 



each step, and using infinitesimally small gradient as- 
cent steps. 

In this paper we presented AWESOME, the first gen- 
eral algorithm that is guaranteed to learn to play op- 
timally against opponents that are stationary (and 
against opponents that eventually become stationary) , 
and to converge to a Nash equilibrium in self-play. 
It has these two desirable properties in all repeated 
games (with any finite number of agents, any finite 
number of actions, and unrestricted payoff matrices), 
requiring only that the other players' actual actions 
(not their strategies) can be observed at each step. 
AWESOME also does not use infinitesimal steps at 
any point of the algorithm. 

The basic idea behind AWESOME {Adapt When Ev- 
erybody is Stationary, Otherwise Move to Equilibrium) 
is to try to adapt to the other agents' strategies when 
they appear stationary, but otherwise to retreat to a 
precomputed equilibrium strategy. 

At any point in time, AWESOME maintains cither 
of two null hypotheses: that the others are playing 
the precomputed equilibrium, or that the others are 
stationary. Whenever both of these hypotheses are re- 
jected, AWESOME restarts completely. AWESOME 
may reject either of these hypotheses based on actions 
played in an epoch. Over time, the epoch length is 
carefully increased and the criterion for hypothesis re- 
jection tightened to obtain the convergence guarantee. 
The AWESOME algorithm is also self-aware: when it 
detects that its own actions signal nonstationarity to 
the others, it restarts itself for synchronization pur- 
poses. 

7. Future research 

The techniques used in proving the properties of AWE- 
SOME arc fundamentally different from those used 
for previous algorithms, because the requirement that 
the opponents' whole strategies can be observed is 
dropped. These techniques may be valuable in the 
analysis of other learning algorithms in games. 

The AWESOME algorithm itself can also serve as 
a stepping stone for future multiagent learning algo- 
rithm development. AWESOME can be viewed as a 
skeleton — that guarantees the satisfaction of the two 
minimal desirable propcrics - on top of which addi- 
tional techniques may be used in order to guarantee 
further desirable properties. 

There are several open research questions regarding 
AWESOME. First, it is important to determine which 
valid schedules give fast convergence. This can be 



studied from a theoretical angle, by deriving asymp- 
totic running time bounds for families of schedules. 
It can also be studied experimentally for representa- 
tive families of games. A related second question is 
whether there are any structural changes we can make 
to AWESOME to improve the convergence time while 
maintaining the properties derived in this paper. For 
instance, maybe AWESOME does not need to forget 
the entire history when it restarts. A third question is 
whether we can integrate learning the structure of the 
game more seamlessly into AWESOME (rather than 
first learning the structure of the game and then run- 
ning AWESOME). 
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